Skip to content

fix(agentx): preserve trace timing and joins - #31

Open
cquil11 wants to merge 9 commits into
mainfrom
cquil11/allow-agentx-trace-idle-cap
Open

fix(agentx): preserve trace timing and joins#31
cquil11 wants to merge 9 commits into
mainfrom
cquil11/allow-agentx-trace-idle-cap

Conversation

@cquil11

@cquil11 cquil11 commented Jul 30, 2026

Copy link
Copy Markdown

Summary

  • enforce --trace-idle-gap-cap-seconds only as an observed runtime idle watchdog per complete trajectory tree
  • preserve source dataset timestamps and end-to-start replay delays without load-time rewriting
  • restore every surviving warmup stream to one flattened per-trajectory dataset clock so non-burst profiling preserves cross-stream order and spacing
  • include the root, subagents, and flattened-agent streams under one runtime-root watchdog
  • preserve deferred child and join state across accelerated warmup handoff
  • release profiling joins only after both the recorded replay deadline and all required child completions
  • keep both per-tree and whole-system watchdogs active when a cap-advanced timer is retained by a replay dependency

Runtime semantics

The trace watchdog starts for initial future profiling work and restarts whenever the final in-flight request across a root and all descendants completes. If no request from that tree reaches the wire before the configured cap, AIPerf uniformly advances only that tree's pending scheduler timers. If the first advanced request is still dependency-blocked, the watchdog continues within the same expired idle interval until an eligible request reaches the wire. Other roots are untouched, relative timer spacing is preserved, and dependency gates remain authoritative.

Accelerated warmup intentionally removes runtime delays. At handoff, timestamped roots, subagents, sidecars, and gated joins are restored to their next position on one shared tree clock. Profiling subtracts one global minimum, so the earliest request starts immediately while all remaining streams retain their original flattened order and relative start spacing. Timestamp-less datasets keep the existing per-stream delay fallback.

The global idle observer is registered when each phase becomes active, so warmup teardown cannot clear profiling's observer. The dataset loader does not rewrite timestamps for the trace cap: recorded timestamps, api_time, and derived end-to-start delays remain unchanged. Actual server completion behavior determines when a runtime tree becomes idle.

Validation

  • warmup handoff tested with 1, 10, and 100 additional requests per lane
  • prior pending-child turn-0 thundering herd reproduced before the fix and eliminated after it
  • exact root/subagent ordering and SPAWN_JOIN completion gates verified through the HTTP mock server
  • dependency-blocked watchdog candidate reproduced; the next eligible timer issued within the same cap interval
  • issue agentic_replay drops post-child delay on parent resume ai-dynamo/aiperf#1231 root-child-root timing shape verified with and without the separate global idle guard
  • independent sibling-stream drift verified against the runtime whole-tree idle cap
  • active-phase observer ownership and initial profiling idle verified with real scheduler timers
  • broad unit, component, adversarial, and real HTTP mock-server suites passed independently by multiple reviewers
  • all pre-commit hooks passed

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@abf55f902cde0a3e8389c452a83ecbf5ba15dea9

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@abf55f902cde0a3e8389c452a83ecbf5ba15dea9

Last updated for commit: abf55f9Browse code

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Fern Docs Preview: generation failed — see the Actions log for details. This does not block merge; ask a maintainer to retry if needed.

@cquil11
cquil11 marked this pull request as ready for review July 30, 2026 16:49
@cquil11 cquil11 changed the title fix(agentx): allow configurable per-trace idle-gap cap fix(agentx): allow configurable trace idle-gap cap Jul 30, 2026
@cquil11 cquil11 changed the title fix(agentx): allow configurable trace idle-gap cap fix(agentx): preserve trace timing and joins Jul 31, 2026
cquil11 added 4 commits July 31, 2026 11:47
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
cquil11 force-pushed the cquil11/allow-agentx-trace-idle-cap branch from 1c315af to deb5421 Compare July 31, 2026 16:47
ajcasagrande and others added 3 commits July 31, 2026 14:07
…ynamo#1228)

When cache-bust owns trajectory prefix isolation, preserve the same prefix across warmup and profiling so warmup primes the cache used by profiling.

Upstream: 9b5f5f7

中文:启用 cache-bust 时,由其负责轨迹前缀隔离,并在预热与性能测试阶段保持相同前缀,使预热能够填充性能测试实际复用的缓存。

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Keep the default profiling grace period unless the user explicitly overrides it.

Upstream: 5ad0816

中文:除非用户显式覆盖,否则保留默认的性能测试宽限期。

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Carry each stream's full recorded delay across the warmup barrier, then subtract one phase-wide minimum so the earliest profiling request starts immediately without changing relative timing, order, or join gates.

中文:在 warmup 屏障后保留每个流完整的记录延迟,再统一减去 profiling 阶段的全局最小值,使最早请求立即启动,同时保持相对时序、请求顺序和 join 门控不变。

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

@ajcasagrande ajcasagrande left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cquil11 added 2 commits August 1, 2026 16:38
Keep one replay origin per active trajectory tree during accelerated warmup. Rebuild every surviving root, child, sidecar, and gated join on that shared dataset clock so non-burst profiling preserves cross-stream order and relative spacing.

Retain the existing per-stream delay fallback for timestamp-less datasets and seed recycled warmup roots with a fresh origin.

Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Register the global drain observer when its phase becomes active so warmup teardown cannot clear profiling ownership.

When a per-root cap advances a dependency-blocked timer, retain the expired idle budget and immediately try the next timer without bypassing replay barriers.

Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants